Insert separator for more token pairs that would otherwise merge#73
Insert separator for more token pairs that would otherwise merge#73Sanjays2402 wants to merge 1 commit into
Conversation
serialize() inserts a `/**/` comment between two adjacent component values
whenever emitting them back-to-back would let the tokenizer re-read them as a
single, different token. The pairs that need this are listed in BAD_PAIRS.
BAD_PAIRS was missing seven pairs from the CSS Syntax serialization table, so
serialize() silently changed the token stream on round trip once the two tokens
were adjacent (e.g. after whitespace was dropped by a transform):
tokens serialize() re-parses as
number + '%' delim '1%' percentage
number + '-->' (CDC) '1-->' dimension '1--' + '>'
'#' + '-' delim '#-' hash '#-'
'#' + '-->' (CDC) '#-->' hash '#--' + '>'
'-' + '-' delim '--' ident '--'
'-' + '-->' (CDC) '--->' ident '---' + '>'
'@' + '-->' (CDC) '@-->' at-keyword '@--' + '>'
Root cause: a lone '-' delimiter merges after '#'/'-' (forming a hash or ident),
a CDC ('-->') merges after '#'/'-'/number/'@' (its leading '--' is consumed as
hash / ident / dimension-unit / at-keyword characters), and a '%' delimiter
merges into a preceding number to form a percentage. dimension+CDC and the
number+ident/function/url pairs were already covered; these seven were the gap.
Add the missing pairs (mirroring the existing spec table) and a parametrized
regression test that drops the whitespace between each pair and asserts the
serialized output still re-parses to the same two tokens.
https://drafts.csswg.org/css-syntax/#serialization
|
Hi! Thanks for this pull request. I’m curious: what’s your use case? We don’t meet people who serialize their manually created list of tokens that often! 😆 |
|
This came from a parse → transform → serialize round trip rather than a downstream production bug report. For example, a transformer or minifier can parse I don't have a concrete downstream package currently hitting this. If preserving manually transformed token streams is outside tinycss2's intended serializer contract, closing this is reasonable. |
Summary
serialize()documents that it "takes care of corner cases ... [and] consecutive identifiers that would otherwise parse back as the same token". It does this by inserting a/**/comment between any adjacent pair of component values listed inBAD_PAIRS.BAD_PAIRSis missing seven pairs from the CSS Syntax serialization table. When two such tokens end up adjacent — e.g. after a transform drops the whitespace between them, which is a routine operation —serialize()emits CSS that re-parses to a different token stream, silently changing the meaning of the input.Reproduction
The two-token list
[<number 1>, <delim %>]serializes to1%, which parses back as a single percentage token.Affected pairs
All are in the spec serialization table but were absent from
BAD_PAIRS:serialize()number+%delim1%number+-->(CDC)1-->1--+>#delim +-delim#-#-#delim +-->(CDC)#-->#--+>-delim +-delim-----delim +-->(CDC)--->---+>@delim +-->(CDC)@-->@--+>Root cause
A lone
-delimiter merges after#/-(forming a hash or ident); a CDC (-->) merges after#/-/number/@because its leading--is consumed as hash / ident / dimension-unit / at-keyword characters; and a%delimiter merges into a preceding number to form a percentage.dimension+CDC and thenumber+ident/function/urlpairs were already covered — these seven were the gap.I derived the exact gap by transcribing the spec's 12-column serialization table and diffing it against
BAD_PAIRS; the diff was precisely these seven pairs, and an exhaustive pairwise round-trip check over all token types flagged the same seven (and nothing else).Fix
Add the seven missing pairs, mirroring the existing spec-table style already used in
BAD_PAIRS(+9 lines inserializer.py, comment included).Tests
New parametrized
test_serialize_adjacent_tokens_do_not_merge(modeled on the existingtest_backslash_delim): for each pair it parses the whitespace-separated source, deletes the whitespace token so the two component values are adjacent, serializes, re-parses, and asserts the token types and serializations are unchanged.Proof the test guards the fix (stashing only the
serializer.pychange):9 failed9 passed16984 passed(was16975), no regressionsruff checkclean on both changed filesDiff is
+34 / -0acrosstinycss2/serializer.pyandtests/test_tinycss2.py.